GE in TREC-2: Results of a Boolean Approximation Method for Routing and Retrieval

As before, there is a tremendous variation in the topic-by-topic results, suggesting that a great deal more research is needed to nd how to get the best results in dierent routing and retrieval scenarios. We are encouraged by the progress of our system, as well as of the overall eld, in these experiments, and are hopeful that in the coming years we will learn how to combine our promising results in corpus analysis with the more mature ranking and retrieval models of some of the other systems. References [1] Paul S. Jacobs. Using statistical methods to improve knowledge-based news method for query construction and topic assignment in trec. 8 This worked surprisingly well in cases where the topic statements were the topic title was a good description (e.g. \Welfare Reform") and absolutely horribly for those with vague titles (e.g. \Find Innovative Companies". We tried to recover from these by including more words from the description and narrative, but then we had to start recognizing the language of these descriptions, ltering out words like \relevant", \mention" and so forth. The fully automatic ad hoc system certainly didn't do as well as the manual routing system, but it was still at or above median for more than half of the ad hoc topics. Considering that this method could be used within the context of most any legacy retrieval system, the result is worth noting. Furthermore, the generation of Boolean queries from natural language descriptions is an interesting , as well as practical, research problem, because many dierent retrieval systems can make some use of Boolean queries. 2.3 Ranking Our document ranking method will be more fully described in the proceedings paper. In both routing and ad hoc, we used a set of word weights, acquired from the relevance judgements in the routing case and from the corpus data in the ad hoc case. We combined the weighted frequency of these terms with an overall count of the number of topic hits per document, normalizing for document length, to produce a score for each document. This was the result of trying many dierent approaches on the test data, so it was denitely a good method for our system. However, in comparing our results with those of other systems, our precision curve across various recall points is not nearly as good as a system that does really good ranking. In routing, we are not …